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Sherlock Is a computer-based su pported practice environment f c( a complex troubleshooting job In 
the ANr Force. This chapter describes the training problem for which Sherlock was developed, the 
principle s behind Its development, and Its implementation. The training problem Is severe and representa- 
tive of a common problem in our high-technology society. People who win fin a position for a brief period 
(four years or leas tar many In this Air force job) carry out a set of routine tasks which are wed supported by 
technology. However, pertodtealy, a breakdown occurs: a novel situation requiring sophisticated problem 
solving for which Ittte support Is avalabte. In many cases, such problems must be referred to an expert, 
but such expertise !s difficult to acquire. Semlautomated, routinized jobs do not afford sufficient 
opportunities for complex problem-solving skills to develop, so their incumbents lack the skin required to 
handle novel problems. Sherlock is an environment in which this missing skill can be acquired for a 
specific troubleshooting job. 

The fob domain Sherlock was developed to raise the level of F-13 Manual Avionics Test Station 
Technicians 1 troubleshooting knowledge. These technicians repair electronic modules that have been 
removed from F-15 aircraft because of suspected malfunction. In their daily work, they follow detailed 
wrlBsn trouble sh ooti n g procedures (part of their Technical Orders) and use a test station . The test station 
is a large (40 fP) system of electronic equipment to which the module being diagnosed can be attached. 
By setting various switches on the front panels of the test station, the airman can quickly perform tests on 
the module. When a test In the prescribed test routine Isolates the malfunction, the Technical Orders 
suggest an appropriate repair for the module. 

A serious problem arises when the test station itself has a malfunction. Now, instead of following 
fixed procedures from a check sheet and making use of the substantial technological support of the test 
station, the airman is much more on Ms* own and must engage in complex heuristic problem solving. 
Because they may be working in the field far from help, these technicians have to be self-sufficient: they 
have to be able to repair their equipment. However, because the test station breaks down only about 
once a month (and even then may be diagnosed and repaired by an expert to minimize downtime), a first- 
term airman (someone in his first four years of duty) gets few opportunities to learn this most difficult skid 
on the Job. The technical training airmen receive before reporting for work is oriented toward the 
conceptual knowledge and skill needed for the routine part of the work-which is all there Is for perhaps 
95% of the time. Yet this leaves the Air Force with a dilemma: its training supposes that test station repair 
will be teamed on the job, but the job doesn't provide the relevant practice opportunities. Sherlock 



1 Direct all correspondence to Alan Lesgold LRDC, University of Pittsburgh, Pittsburgh, PA 15260. 
Phone 412/624-7046. 



2 To simplify exposition, we use masculine pronouns. Most, but not all, of our trainee subjects we'e 
male. 
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dots (Nichols, Pokomy at al., hi press). Less experienced technicians who have practiced on Sherlock 
About 20 to 25 hours compare in their a Witty to troubleshoot test station failures with colleagues who have 
four more years of on-the-job experience. 

Sherlock demonstrates a relatively speedy and efficient approach to teaching performance of 
cognitive tasks where: 



1. There is a huge problem space, !.e., there are many possible actions that can be taken, many 
choices to be made in solving a problem. 

2. The Job Is difficult largely because of Its complexity rather than because of some specific 
concep t ual barrier. 

3. On-the-job apprenticeship dealng with the difficult aspects of the )ob is not avaHable. 

ihm m|yn gjjhejofa. To understand Sherlock, It Is helpful to understand a bit more about the job 
for which It was developed Routine troubleshooting generally is organized around a decision tree (the 
service manual for most automobfles contains several such trees). At each branching point In a decision 
tree, one gets a piece of Information and then makes a choice. Eventually, the choices lead to a repair 
action. If that action is successful, the task is completed. A decision tree can be generated any time that 
one can stale a rule for delecting each poesMe fault In a system, where the rule specifies the conditions 
under which that fault Is present and each condition can be verified by a measurement The decision tree 
simply represents an organization of those rules that attempts to minimize the number of conditions that 
need to be verified before the fault is found, it is a labor-saving artifact 

The test station Is another labor-saving artifact Gathering data takes time. When the data are to be 
gathered from electrical tests, time is taken up preparing test equipment reconfiguring the system so that 
the n eede d test points are exposed, attaching any required electrical sources, and attaching Instrument 
probes. The test station minimizes these time costs by acting as a large switchboard. By pushing a few 
buttons and turning a few knobs, a technician can configure a test circuit, specify a measurement to be 
made, and produce the vakje of the measurement on a display. Every test has the same basic form: an 
electrical circuit Is created In which there is a stimulus (a source of patterned electrical energy; perhaps a 
power supply or a signal generator), a device being tested, and a measurement device. In essence, the 
technician Is measuring the effect, at some point(s) in the device being tested, of applying a signal to 
some other point(s). 

When a test "tails 1 ' - that Is, when a test result is out of the expected range - this suggests with high 
probabMty that there is a malfunction in the part of the device being tested. The Technical Orders ted what 
to replace for any given test failure. However, sometimes the mandated repair does not change the 
situation. This generally means that the intended test circuit was not properly created, the difficult 
situation that Sherlock deals with. Either something is wrong in the test station, or something Is wrong in 
the cabling between the test station and the device being tested. A false and possibly misleading test 
value results. If a technician understands what test configuration Is not being achieved correctly, then he 
has greatly constrained the troubleshooting problem. He need only consider the circuit path through the 
test station that Is created for the test that tailed, plus the switching circuitry that is Involved in producing 
that configuration. This is because the test station is a giant switchboard. Only the circuit created by 
particular switching, phis the switching circuitry involved in creating that circuit, need be considered when 
a circuit tab. 

So, an airman technician must learn several things to successfully diagnose test station failures. First, 
he must be able to figure out what test the station was attempting when it failed; he must have a mental 
fllfiufil of the test. From this, he can determine which circuitry, distributed throughout the test station, 
might be the locus of failure. Then, he must devise a altalsa* for searching this circuitry for the fault. This 
wM involve narrowing the search space by performing various tests. For example, if no signal even gets to 
the device being tested, then none can get to the measurement module. In that case, the problem must 



be between the signal source and the connections of the test station to the device being tested, not In 
the path from the device to the measurement module of the test station. 

De ve loping the strategy win require some understanding of what a test is, how the test station carries 
out tests, and how the major components of the test station work. Carrying out a casting strategy will 
require repeated exercise of certain skills of measurement and of circuit diagram Interpretation. So, in 
addition to a mental model and the right strategies, the technician must have the procedures needed to 
follow schematic diagrams and to make and interpret tests with various instruments such as an osclio- 
scope and a multimeter. 

Note that a strategy can be systematic but uninformed. For example, one could replace components 
of the test station one at a time until the bad component was found. TMs is inefficient because the circuit 
path Involved In a test tends to Include a little bit of each of a large number of components. However, 
serial component-swapping approach Is widespread among electronics technicians in general, even 
though Is usualy hopelessly Impractical. 



Plnclples Guiding Sherlock's Development 



In building Sherlock, we were constrained by several principles. Additional constraints arose from 
the projects status as a prototype effort. In this section, we discuss why Sherlock Is the way it is. 
Sherlock Is a practice environment; it affords chances to practice doing a task !n a supported context. At 
the most abstract level, It is the cognitive version of earlier approaches to errorless learning (Terrace, 
1964). With support from a computerized coach, the technician successfully completes problems of the 
complexity he wants to learn to handle on his own. 

The role of practice. Rather than calling it a tutor, we refer to Sherlock as a su pported or coached 
practice environment The fundamental purpose of Sherlock is to provide an efficient artificial means of 
practicing, with support and feedback, a skill that cannot be practiced very efficiently In its real application 
context The routine activity of the menu* avionics test station techniclai is to foltuw an essentially fixed 
procedure. Specific rules are read from a document and followed; there Is little Judgment required. On 
the other hand, when the test station fals, a strategy must be invented on the spot--a different strategy 
each time, because the failure Is different each time. Verbal principles have to be mapped into richly 
varying sNuaHons in order to carry out this work. Practice Is the means for making initially verbal knowledge 
stronger and mors flexible (Anderson, 1983). 

What mates padfifl *i™» effective? For practice time to be effective, several things are required. 
First, the proportion of time that involves the target cognitive activity should be maximized. If time is spent 
waiting for parts, dismantling components, and carrying out long sequences of performances that are 
already well learned, then that time is not available for practice of the decision process that is the core of 
troubleshooting facility. Second, cognitive overload should be controlled. In a complex cognitive 
performance Ike troubleshooting a test station failure, a trainee may simply forget what he has done so far 
or where ho is In carrying out a plan. WNIe trainees need to learn to handle this complexity, we assume 
that this virtuosity w* come with practice. So, we provide help wten a trainee loses track of what he has 
done. This minimizes the experience of Inadequacy and saves the time It would take to declare a failure 
and stage another problem. Often, simply being reminded of what he has already done win be sufficient to 
enable a trainee to go on to work on a problem effectively. Third, advice should be available to overcome 
knowledge gape. If the trainee simply does not know what to do or how to proceed, giving him a hint will 
both teach the missing knowledge and enable overall problem solving practice to continue. Fourth, 
performance should be coached beyond success toward optimality. Just as an athletic coach can watch a 
complete performance and then improve it by providing useful insights to an athlete, so a problem solving 
practice coach should provide critical insights that move the trainee toward more efficient performance. 



Refining the environment rather then refining the teacher While coaching is an important aspect of 
Sherlock, primary emphasis is on the practice environment not on the role of coach. This is a difference 



In nuance, not an absolute diffbrence, but it means that we are focused primarily on having the environ- 
ment reveal Re structure rather than on having the coach know exacfly what to say to a particular student. 
Thte approach Includes certain biases. For example, the trainee Is given much of the responsibility for 
learning. Very Ittle feedback is provided other than that which the work environment normally provides; 
help must be requested. The assigned tasks, though at the Hmtts of trainees 9 capabilities, must all be 
com pl et e d, with support from the environment available on demand. Fidelity to the work environment is 
emphasized over precision of student modeling. Our view is that Sherlock can know exactly how the 
solution of a problem is proceeding but has only imperfect data about what a student knows, because 
Inexpert perf or mance Is inherently unreliable. 

The contaxf -apedflc abstracted problem space . Diagnosing a test station faflure Is a big problem, 
because of the many components the station contains. Having the right mental model constrains the 
problem, but it is sM extremely complex. Sherlock's r roblem spaces 3 are computationally manageable 
only because several techniques have been used to make them smaller (cf. Lesgdd, Lajoie, Logan, & 
Eggan, In prees). First, they do not contain al the theoretically possible comofnations of actions. They 
contain only actions tttat actually tend to be taken by trainees or experts. A number of actions that might, 
In principle, be taken are left out simply because our experts assure us that they never occur. Second, a 
separate problem spece Is developed for each problem. While these spaces have a lot in common, the 
space tor one problem is much smaller than the space of al posst>ie test station failures, because only a 
•me* part of the teet station is Involved in a given test and the context of failure is tf it some test of an 
avionics device yielded spurious results. By eliminating actions that are not seen In real Hfe and by 
specMizing the problem space to a particular problem context, the space can be made much smaller, 
though It also becomes more contexts peclfic. 

The third technique used to simpNfy the problem space Is abstraction. For purposes of representing 
the problem space, we do not consider each of the test station's individual electrical components, of 
which there are thousands. If we did this, just representing the posslbWy that each one of the thousands 
of parts could be broken would make the problem space unmanageably large. So, we abstract the test 
station structure to the levei of replaceable modules, such as printed circuit boards, rather than represent- 
ing each component on each board separately. As a result, our context-specific abstracted problem 
spacse have 50 to 60 nodes, rather than the thousands of nodes that a "complete" problem space for test 
station dagnools would have. 

An oMacfcfldaOiad approach. The individual test measurements that trainees make whHe diagnosing 
statton falures are measurements at specific points within a circuit not generic measurements of a who!e 
circuit board. This Is handled by having a specialist miniprogram for each node of the abstracted problem 
space. Computer software designers call these specialist minlprograms objects. A generic template 
object Is first defined, and then each node's "personal" object is specified as a specialization of the 
generic template. Most of the specialization consists of data, not program: specific hinting information, 
test point values, etc. 

Substantial savings result from this combination of an abstracted problem space with specialized 
objects to bring each abstraction back to the real world's level of detail. Rather than a network of separate 
problem space connections between every detail of one module and every detail of each of the others, 
there Is only one path to or from each abstracted object The approach is similar to the pattern of traffic in a 
large city. The abstracted problem space Is like the network of main streets, and the objects are like the 
side streets of smal neighborhoods. One gets from a neighborhood to a main street, takes the main 
street to another neighborhood, and then takes side streets to a specific address. Separate roads 
connecting every pair of addresses would be unthlnkably more complex. 



3 As noted in the introduction to this volume, a problem space is a network whose nodes represent 
states of partial problem solution and whose links represent transitions between those states. Thus, any 
solution process can be represented as a path through the problem space from the problem as originally 
posed to a solution. 
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The goal* of training: Curriculum. So far, we have discussed properties of the problem space. A 
separate concern Is the curriculum, the rat of instructional goals and the sequence of activities that are 
meant to accomplish those goals. Sherlock is a holistic practice environment. What trainees do with 
Sherlock is to solve difficult test station diagnosis problems. Each problem exercises many of the 
needed skH components in the context of a complete, naturalistic performance. Tnis is very different from 
many instructional activities, such as courses, where different sessions treat different currlcular goals. In 
Sherlock, every session addresses almost every currlcular goal. Nonetheless, Sherlock does have an 
expldt representation of the instructional goal structure it wants to achieve. The three main areas of its 
goal structure, or curriculum, are (a) troubleshooting strategy, (b) mental models of test configurations that 
can be created with tie test station, and (c) using test instruments to make measurements. 

ModeHng the atudenfs emerging general competence and problem specific performance . 
IntaMgont training systems generally maintain some form of student model. Tutors that deal with 
constrained bodes of knowledge, like proofs of geometry theorems, model the student by trying to 
define a computer program that wH behave just as the student behaves. A comparison between that 
program and a program hat simulates an expert reveals what the student has not yet learned and is the 
basis for Interactions with the student Sherlock does not use simulation techniques to model student 
performance. nscMgnoses are maoe m a simpler way. 

Baslcaty, Sherlock keeps two kinds of student models for each trainee. 'A competence model is 
maintained; this is a notation of how wen each currteutar goal has been achieved. A performance model Is 
generated tor each problem that a student attempts to solve. It is an annotation of how weH the student is 
expected to do at each point of the abstracted problem space. The object (microprogram) corresponding 
to each node of an abstracted problem space has, as part of Its local microdatabase, information about 
which currlcular goals are relevant to Its execution. 

Developing a predicted performance model for a student on a specific problem amounts to 
determining, for each nock of the abstracted problem space, how well the student has mastered the 
currlcular goals relevant to that node. The performance model then influences how reticent Sherlock will 
be In providing expldt help at specific points In the problem space. After the problem is solved, any 
divergence of student performance from the predictions can be considered in updating the student's 
competence model. So, If a student does better than expected at son* point In a problem, this is prima 
fade evidence that he has improved on the currlcular goals relevant to that point 

An Example 

The domain targeted by Sherlock and its mode of operation may bp jo me clearer through an 
example problem. One of our easier problems involves a situation in which , whHe a device from an airplane 
is being tested, an Incorrect meter reading appears on the test station. The documentation available to 
the technician asserts that repladng a particular part of the device will result In restoration of the correct 
metsr reading. When this replacement fails to eliminate the problem, the test station itself becomes a 
possfole suspe*. An expert wHI realize that only a small portion of the test station is involved in the test 
measurement that failed. WhHe the whole test station requires a few hundred pages of schematics to 
capture all of Its complexity, an expert's mental representation of the portions relevant to this problem 
would fit Into one page, 4 shown as Figure 1 . 

[Figure 1: Mental Model of Expert] 

Acmes the top of the Figure, the path is shown from the device being tested (UUT, for "unit under 
test") to the dgital multimeter (DMM) on the test station. From left to right, the circuit path passes through 
connec to rs (J1/P1) to a cable (TP, for "test package") and then through additional connectors (P2/J2) to 
the test station. Everything else In the diagram is part of the test station. Upon entering the test station, 
the signal from the UUT passes through three printed circuit boards before reaching the meter (DMM). 



4 An extensive task analysis (Lesgold, Lajoie et a!., 1986) supported our assumptions concerning 
expert domain knowledge and performance as well as the ways in which nareand less astute technicians 
hande test station troubleshooting tasks. 
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Theee are labeled A1A3A2, A1A3A1, and A2A3A1; the codes reflect three-dimensional coordinates in 
the 40 (I s test station. Each of the printed circuit cards is complex enough to require a page or more of 
schematic (flagrant. In Figure 1, only the relevant connector pins (numbers around the edges) and the 
path the signal Wees through the card are shown. So, for example, the signal enters A1 A3A2 through 
pins 57 and 58 and exits through pins 11 and 12. The top part of Figure 1 represents ttv* signal flow 
aspects of the problem. However, the signal passes through several relays, which are controlled by 
switches on the front of the test station. The circuitry Involved in translating switch settings Into a signal 
flow path la called the dataflow portion of the problem, since data entered as switch settings are translated 
Into a particular station configuration. Four additional printed circuit cards, A2A3A7, A2A3A8, A2A3A9, 
and A2A3A10, are Involved in data flow. Several switches, S52, S53, and S33, generate the signal path 
control data for this situation by being either open or closed. 

An expert solution to the problem Is ■ustrated by the dark ellipses. The expert might first check at 
the P2/J2 connectors to see if the voltage value expected on the DMM is getting Into the test station In 
the flrst place [Stop 1]. H N Is, then he might check halfway between P2/J2 and the meter (Step 2J. In this 
problem, the signal is found at Step 1 and is missing at Step 2. This Implica tes card A1A3A2. The expert 
would verify fie card's input at phis 57 and 58 (Step 3]. Since the signal Is getting Into A1 A3A2 but not 
getting Into the next card, A1A3A1, a Ikeiy problem is that the relay that switches the signal through 
A1A3A2labad. Step 4 of the expert Is to test that relay to see If the actuating voltage Is passing through It. 
Since It Is not, the expert traces through the data flow path and notes that the actuating voltage should be 
coming from A2A3A10. He then checks to c*ee if the correct control voltages are reaching that card (Step 
5J. They are not So, he testa to see if the switch that controls those voltages Is working (Step 6]. It 
shows a short In a part of the switch that should be open (Step 6]. So, the expert swaps the switch (Step 
7], and this solves the problem. 

In malty, this actMty involves perhaps a square yard of complex schematic diagrams, so It is nontrtvtai. 
In fact, our trainees generally were far less facile than the experts In solving this problem. For example, 
while our expert solved it in seven steps, one trainee took 14 steps and another took 20 (the others were 
shnlar). Figure 2 shows the activity of the trainee who took 14 steps. Superficially, It Is apparent that he 
tended to trace through circuit paths from one end to the other rather than trying to Isolate the problem to 
one half or the other of the remaining suspect region (we call the expert approach "space spRtting"). Abo, 
he Med to make a few measurements that were not on the circuit paths Involved In the failure con- 
figuration. Some redundant teats were also done (e.g., Step 6 vs. Step 8). However, he did solve the 
problem. 

{Figure 2: Mental Model of Trainee] 

While performing tests on card A1 A3A2, our example trainee asked for help several times. First he 
was just given an overview of what he had done so far. Asking for additional help, he was told by 
Sherlock: 

You need to determine if the 0.0 ohms from the UUT Is getting to this 
card. If there Is an input you need to test the output. If there Is no output 
you need to determine why the 0.0 ohms signal Is not getting through 
this card. 

Asking for even more help, he was given more specific Information about which pins on the card were 
relevant After doing the required measurements, if he had asked for more help, he might have been told: 

The Input to the A1A3A2 is good. The output from the A1A3A2 is bad. If 
there's no output from this card, then you should conclude that this card, 
or the data to this card which sets the relays, is the cauae of the problem. 
Since A1A3A2 Pin 36 reads +28VDC, when it should read OVDC if 
TPA03 Is to be set, Sherlock suspects that you should Investigate the 
A2A3A10 card, pile failure was In a switch, but the A2A3A10 card Is the 
next sensible place to look at this point in the problem.] 



6 8 



The iralnM In this example tried, on about a half dozen occasions, to take an unsafe step, usually 
removing a card without first turning off power to it. Whenever this happened, he received an appropriate 
warning. He also tried to make some measurements with the meter set incorrectly and got feedback about 
that problem. 

NT* that an important property of Sherlock is that it affords opportunities for practice of the 
complex ski and that It speaks primarily when spoken to. In certain respects, K is a rich environment, and 
the student not the computer, is the teacher and is in control. Sherlock exerts control where the 
environment would, on issues of safety or impossibility. It also tries to cut off long ventures down dead 
ends, but for the most part the trainee decides what steps to take, when to seek advice, etc. We think this 
promotes active, conscious learning, and trainees have been willing to take the responsibilities that 
Sherlock implicitly assigns to them. 

Implementation of the System 

Overview of the Design Approach 

Sherlock's knowledge base has three components: the work environment, the abstracted problem 
space for each problem, and Ihe uirricukm 

Thawofkanvfaonmant. Sherlock presents a partial simulation of the work environment in which the 
skMs It te aches are ordnarty exercised They <rob of the test station are displayed and are manipulable, 
and control settings art rnormored so that ua , or inappropriate actions can be pointed out and blocked. 
Since the basic data-gathering action of troubleshooting the test station is to measure electrical properties 
at various points in the circuitry, Sherlock allows trainees to make measurements by pointing to the spots 
In a schematic diagram at which meter probes should be placed. 5 In addition to the measurement devices 
buiR Into the feet station a hand held multimeter Is also simulated as an available device that can be 
appled to the schematic diagrams, and so is a wire, since many tests can be performed by shorting across 
various points and observing the effects on overall test station function. 

The schematic dtagrams are taken from the Technical Orders, printed manuals used by technicians In 
their everyday work. There are, of course, other components to the Technical Orders besides the 
schematic diagrams, Including indexing of the schematics by test point and components structure. 
There are also extensive troubleshooting procedures, for diagnosis of units fra,i the aircraft using the test 
station and for dtagnosis of the test station itself (including the lest package" that connects units to the 
test station). 

Tha abstracted pmN^fipflifr If we simply had the trainee carry out tests by pointing to schematics, 
inferring his strategy and tactics would be difficult. Also, there are a number of special acts, such as 
tightening connections, swapping boards, and replacing specific components, that cannot be repre- 
sented by pointing to schematics. For this reason, a rich hierarchical menu scheme is used as the basic 
means whereby a trainee traverses the problem space. At any given moment, the trainee can, by 
appropriate use of an action menu, continue movement through the problem space. The data structures 
to support this are feasible only because we can specify an abstracted problem space that is small enough 
to be managed (because of constraints based on the problem itself and the stereotypy of trainee and 
expert performance). In practice, we have found the abstracted problem space Idea to be workable, and 
we expect that ex tens i ons to the approach win not require a major change. 

Connections between the abstracted problem space and tha other two components . There are 
loose couplings between the abstracted problem space and the other two components of Sherlock's 



5 As wH be discussed below, not al test values are available. That is, our work environment simulation 
is not as complete as It could be (compare to SOPHIE [Brown, Burton, & De Kleer, 1982), for example). 
The strategy for deciding which data values should be available is a subject of continued research on our 
part 
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data truck**, the curriculum and the work environment The connection between the curriculum and the 
a b s tracted problem space makes possible the primacy means of Individualized coaching In Sherlock, a 
rich hinting mechanism. In the current version of the tutor, each student encounters the same 34 
problems, which take an average of 35 minutes each for them to solve. The primary mode of kKftvkfuaiiza- 
Hon over the average 20 hours of coached practice Is the hints. The average student sees abou* 35 hints 
during the 20 hours, and these are highly particularized according to where in the course of problem 
solution he was when a hint was requested and how well the object corresponding to that point in the 
abstracted problem space expected him to perform the next required step. 

This hdhriduaRzation is possible because each node of the abstracted problem space is represented 
by a computational object that knows which curriculum goals must be accomplished before a trainee can 
handto the part of the problem space R represents. As noted above, this allows the object to construct an 
expectation of the level of trainee capability for Its part of the problem space, using the curriculum based 
student competence model for that trainee. Each problem space object dsn knows about a variety of 
hints, both prepared hints and ones that can be constructed for the specific c iston. The choice among 
hints la determined partly by the specifics of the student's recent activity and party by the object's 
performance oxpoctaMon for the student 

The connection s to the work environment have to do with individual tests the airman may want to 
cany out by pointing to various points in schematic diagrams, in these Interactions with the work 
environment the airman la realy taking active problem solving steps, so It is necessary for the work 
•nvironment to, In some sense, be coupled to the abstracted problem space action menu scheme. This 
would be easy if the step taken by the airman was always exclusively a sensfcfe next step in his trouble- 
shooting plan. However, it Is quits posstole for an akman to make additional measurements on a particular 
board just because he has current access to it* We couple the work environment to the abstracted 
problem space by having small production systems 7 tied to each work environment unit (each schematic). 
When the airman finishes testing on a schematic its production system is run to notify relevant abstracted 
problem space nodes that information relevant to them has been collected. 



The Problem Space Repreaentatlon 

As noted abjve, the trainee's problem solving activity on Sherlock Is monitored and guided by a 
representation of the abstracted problem space for the problem currently being performed. This 
ab s tracted problem space reflects a range of both situation-specific and more genera' knowledge. This 
Indudee the architecture for the test station, the function of the tests that can be carried out with the test 
station, and various strategies, In either weak (generic) or strong (domain-specific) forms. Each of these 
knowledge components constrains the problem space In important ways. 

tafllMDflM of torn structural model of tna test station The work environment consists of the test 
station, the unit from the aircraft that Is currently being tested, and a test package which connects that unit 
to the test station (the station is somewhat generic, and the test package contains the specializing, 
sometimes active, circuitry that enables K to test the specific aircraft unit). Within the test station, there are 
twelve large drawers - power supplies, signal generators, switches, and measurement devices - 



• As wH be discussed below, not al test values are available. That Is, our work environment simulation 
is not as complete as it could be (compare to SOPHIE (Brown, Burton, & De Kleer, 1982), for example). 
The strategy for deciding which data values should be available Is a subject of continued research on our 
part 



7 A production Is a rule that calls for some operational step to be taken if a particular set of conditions 
Is met A producdon system is a program that consists of a set of rules. The program Is executed by 
repeated^ checking to see which rotes 1 conditions are satisfied, executing their actions, and then 
recydng through the condition-checking process. 1 n 
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con f ining too various components thai enable testing of aircraft units. Figure 3 shows how this structure 
to repr es ent ed on the computer screen In Sherlock. 

To a large extent, the dr a wer s represent a first cut on a functional decomposition of the test station. If one 
were toM onty that the test station contained a defect, it might be reasonable to try to localize the defect 
first to a specific drawer and then to a specific circuit board or other component (e.g., transformer) within 
the dr a wer, Just as a physician's first effort to narrow down a diagnosis might be to decide It is a problem of 
the cardiovascular system, or the digestive system, or some other system, each of which is in somewhat 
sequestered anatomy. 

[Figure 3: Sherlock Basic Screen] 

Injiaoeai °* lha ****** model of a in fact, though, the real diagnostic situation Is much more 
constrained. Test station failures become manifest when the test station Is used for a purpose and that 
purpose cannot be achieved. We have designed Sherlock to emphasize a fundamental mental model, 
the mental model of an electronic test In Its generic fonrt, a test consists of four components, a stimulus, a 
device being tested, a load, and a measurement device. The stimulus is a source of patterned energy that 
passes through the device being tested to the measurement device, with the load being used to 
d h sl p a to excess energy or to transform the output signal Into a pattern within measurable range. 

[Figure 4: Basic Structure of a Test on the Test Station] 



Figure 4 helps clarify the core mental model. For any given test that the test station is carrying out on 
a place of equipment from an airplane, a signal (perhaps one or more voltage levels, perhaps something 
as complex as a radar signature) Is generated by the test station and routeo through a switch (caHec the 
switching complex or RAG drawer; more or less Hke an old fashioned phone exchange) to particular circuit 
pathe in the teet package and thence to the unit being tested. Various outputs of the unit being tested 
are routed through the tesi package to the switch which then routes them to a measurement device In t 
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test station. 

The Idea is that me test station configures Itself to carry out tests. When it fans, tt fans in the course of 
carrying out a teet The falure must be somewhere In the particular configuration of the four components 
of the teet (stimulus, device being tested, load, and measurement device) or in the paths between them. 
Coneequentty, efficient troubleshooting should be organized, in part, as an effort to identify the parts of 
the test staton that are Ming the four rolee in the test that failed, to Identify the pathways in the test station 
that connect thoee components, and then to spilt the space of thos^ Identified components into 
subepacoo that can be efficiently determined to either Include or not include the system fault. Further 
complexity arises because the real model of the test station must Include the switching device that 
implements connections among particular components to realize a particular test. That Is, the paths 
between the components of a test are created by other test station components, the switching logic. 

MuanQM Of troubleshooting strategies. Finally, the abstracted problem space has to take account 
cf the trouWeehootfng strategies that novices and experts win employ In trying to find a fault. Basically, 
there are two types of strategies, space splitting and generic cures. By a generic cure we mean a step one 
can take when no systematic plan Is in hand or apparently productive. For example, sometimes a 
connection will loosen. So, tightening a number of connections will sometimes fix a test station. 
Tightening a eat of connections would be considered a generic cure if it was done without being 
motivated by the process of searching the problem space systematically for a solution but rather as a 
dyfauft. Tils has some chance of fixing the problem. However, generic moves often are a form of 
thrashing around by novices who do not know what to do. 

More Interesting are the space splitting variations that we see in domains like the one we have 
studied. In our samples of airmen who have been through electronics training and who were selected to 
have high promise of being able to do their Jobs, almost ad try to be systematic either by space splitting or 
by moving sequentially through a circuit. What differs with expertise is the space that is split when space 
splitting strategies are employed. Some split the space of swappable drawers, prtnted-cfrcult cards, and 



other c omponents, while others spHt the space of functional subdivisions of either the test station in 
general or the specific test configuration that failed. Even when spotting the space of a functional mental 
modal, detail of the situated environment play differing roles in space splitting strategies. We have seen, 
for example, eases where the space that is split Is the model of the test that failed but where the 
subdivisions are owappabte cards. This would work well If the swaps could be doni qutekiy, but in realirte 
(as opposed to our tutor), unmotivated swaps are very Inefficient compared to making a few measure- 
mania and figuring out which specific component should be replaced. 

We do not completely capture these strategy considerations in our tutoring, but our abstracted 
problem spaces seem sufficient to allow them to occur and to be noticeable. We 3ee little evidence that 
there were airmen who could not try what they wanted to try in problem solving because of some missing 
pieces of the problem space. This is partly because the mechanisms that allow specific tests to be done 
on particular test points of a particular printed circuit board ore part of the work environment simulation and 
not part of the problem space representation per se. At the more microscopic level of placing meter 
probes anywhere on any of our 50 pages of schematic diagrams, there were occasional cases in which the 
trainee could not do exactly what he wanted to.* 

[Figure 5: Abstracted Problem Space for a Sheriock Problem] 

What do— lha abstracted probtom upaca tor a nmhlam took Hka ? Figure 5 Shows a graphical 
representation of the abstracted problem space tor one of our problems. Each label represents a node hi 
the abstracted problem space, and the Hnes represent hierarchical relationships; nodes on the left 
subsume nodes on the right to which they are connected. That is, the nodes branching off to the right of 
a given node represent the wherewithal for "completing" the purpose of their "parent." This does not 
mean that every "offspring" of a node must be exercised before that "parent" is completed, though. 
Sometime s the actions and outcomes associated with a single "offspring" make It clear that a whole region 
of the problem space can be discounted. 

Each node of the abstracted problem space is a computational object with specific data and 
procedures for handling the variety of circumstances that involve the part of the problem space it 
represents: 

(a) how to keep records of a trainee's activity when he reaches that part of the problem space; 

(b) how to provide hints to the trainee when he reaches that part of the problem space; 

(c) how to take account of the possibility that action elsewhere in the problem space may have 
ruled out this objects part of the space; and 

(d) how to assure that actions taken in this object's part of the space meet certain requirements 
for safety, efficiency, and possibility of being performabie in the real world. 

Table 1 shows an outline of what one of our abstracted problem space objects looks like. 



8 We dW find that on some occasions an airman would have no trouble moving in our abstracted 
problem space but would not always be aOowed by the tutor to perform the exact test that he wanted. This 
was party an artifact We required airmen to specify which board they wanted to test before going to the 
? ^nutation to place the probes of their simulated meters. Sometimes, after doing one test, an airman 
would try to do another on a part of the schematic currently being displayed that was not part of the same 
board (board outiines were shown In the schematics, of course). This was not accepted by the tutoring 
system. A more serious problem was that in a few cases airmen tried to do tests that had not occurred to 
our experts as being likely to occur. This problem Is fundamental with respect to the adequacy of 
abstracted problem space approach and is addressed later beiow. In future versions, we expect to do a 
better job of preserving s sound abstracted problem space approach at the more detailed level of the work 
erwironment tfrnutatJon as weH as at the menu-driven problem-action-step level. 

O 10 ip 

ERJC xc 



[Table 1 about here] 



Why hiMi a aHMMtft problem apace representation ? As one would suspect and our data appear to 
confirm We, there Is much more ktiosyncracy at the level of Individual electronic tests or measurements 
than at the level of the replaceable components such as printed circuit cards. Thus, the design approach 
of (Hiding Individual problems around an abstracted problem space representation but situating the 
activity of the trainee In the work environment has Important practical advantages. For the level of detail at 
which we have good understanding of expert and novice behaviors, we can be very top-down and plan for 
AOwwewMwanttoax performance, attending to both glcbdi and more local detaHs of expert problem 
solving within the domain. At this abstracted level, coaching can be an intelligent process. For the 
microscopic level at which our Incomplete understanding and combinatorial complexity make this less 
poesMe, we can stM provide a rich device simulation and a .we symptom-driven and preprogrammed 
level of coaching. Wit) this two-prong strategy, we feel that we can provide a very cost-effective approach 
to tutor design, brfudtng bUelgent coaching where this is feasible, and can retain maximal situated detail 
insure transfer 10 vie real joo environment. 

The Work Environment Representation 

The work environment has several components. First, there is the representation of the external 
work en v ironment, Including the front panels of the test station, the test package, and the unit from the 
airplane thai Is being tested on the test station. When the display shown in Figure 3 Is on the screen, the 
front pane! of any of the twelve major test station components can be accessed by selecting (with a 
mouse) the appropri a te one of the twelve boxes shown Inside the test station representation. For 
example. Figure 6 shows the front panel display for the oscilloscope drawer. Each of the knobs and 
switches Is manlpulable. Mousing 9 alongside any knob or switch causes it to reset to the position the 
mouse It pointing to. The effects of the change are propagated to all displays Immediately. So, for 
example, the Channel 1 Volta/Divtalon daptay In the top center of Figure 6 could be changed from 1 V/DIv 
to 2/Dlv by mousing the 2 lust below the current setting of 1. The result of doing this would be to change 
the scale of the dteplay, causing the waveform shown on the left to shrink to half Its current size. 

(Figure 6: Oscflloscope Front Panel Display] 

A sir »nd part of the work environment simulation consists of schematic circuit diagrams for many, but 
not aM, portions of the test station and test package 10 These diagrams are also available to the trainee In 
printed form in a booklet that is at the computer workstation for use during practice sessions on Sherlock. 
The reason for putting them on the screen is to provide a highly real-worid-situated environment for 
practice. Trainees make measurements on the test station by calling up a schematic and pointing to the 
test points in the schematic at which meter probes should be applied. 

[Figure 7: Making a Test on a Schematic] 

Figure 7 shows part of the screen during such a test. The hand-held SimpsonTM multimeter is 
indteated in Icon form on the upper left of the Figure. By first mousing the word Bad and then mousing a 
point In the schemed diagram, the trainee can indicate where the red probe should be applied. The icon 
for the red probe is actualy shown In Figure 7, near the middle of the schematic diagram, between test 
points numbered 52 and 20 (it is covering the test point to which it Is applied), it Is also possible for the 
trainee to access the front of the hand held meter to switch it to different ranges and measurement typ es 
(ohms, DC+ volts, DC-volts, or amps). 



9 By mousing, we mean pointing to an object and then pressing a button on the top of the mouse 
(pointing device). 



10 The test package is the connector between the test package and the aircraft device that is being 
tested. R consists of complex caMng and a smaN amount of circuitry. 
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There are about 60 schematic diagrams available on the screen. An extensive index is available to 
the trainee both In print form and actively on the screen (mousing an index entry causes the figure it 
names to be displayed). In addition, cross-reference tables enable the system to know In which 
sc h ema tic s a given last point can be found (In some cases, there are multiple views, and the same test 
point may be in several sc he ma tics, so keeping track of what the trainee Is doing requires this kind of cross 
referencing). Thus, the printed documentation used on the job is wen represented in the work environ- 
ment simulation, s im u ltaneously also allowing a view of the innards of the test station and test package 

The final aspect of the work environment, already mentioned, is a collection of measurement devices 
for testing hypotheses about the location of the fault in the test station. One device, the hand-held meter, 
is quNe an obvious choice. In addition, the meters In the test station itself can sometimes be used to make 
measurements on other dr a wer s . However, In order to capture the forms of activity in the real job 
environment, another device Is also provided, a wire. This Is because R is often more efficient to short out 
two test points and observe meter readb>gs on the test station than to take a hand-held meter and trace 
through a drouH The wire as a "measurement device" Is another example of our extensive efforts to 
capture the cognitive situation of the real work environment as much as possible. Figure 8 shows the 
screen after the airman has received hints concerning a move that uses the wire—he ends up performing 
the test and tt>en asks for and receives additional advice on what the results mean: "This means that the 
measurement petti to the DMM [<NgNal multimeter drawer of the test station] Is bad." 

[Figure 8: Screen Showing Measurement Made After Getting a Hint] 



implementation of the Work Environment Representation 

Drawers and teat equipment. The external appearance of the work environment primarily involves 
the front panels of the test station and the test equipment available for troubleshooting (e.g., the hand- 
held Sknpeon™ meter). Each of thoee panels Is represented as a computational object. The object for a 
drawer or piece of fret equipment contains a variety of variable "slots" that store the state of the object and 
also Information about the trainee's use of the object (currently, we store the time and nature of every 
knob movement in the appropriate object; tests (placements of meter probes) are stored separately, as 
d te c uss od below). Other slots contain the graphic display detals for the background of the object, which 
remains static, for meter displays, which can change value, and for knob settings, which can also change. 
Information about how to configure a small iconic representation is also stored for drawers that have meter 
d is plays. 

In addition, there is associated with each display panel a Hst of active value regions, screen .eglons 
that respond to mousing by changing the display. When a trainee mouses an active region In a drawer 
display, the change associated with that active region is executed. A common change is to move the 
pointer on a knob to the location moused, which also involves propagating the new knob setting value to 
other display components that might be affected by it (e.g., turning the knob that sets the scale of the 
osdHoscope requlree not only changing the knob display but also updating the oscilloscope waveform 
dtoplay). 

Each drawer or test equipment object contains specifications, called methods,, for carrying out certain 
procedures. These Include methods for updating knob settings and display dials, building the original 
dtoplay of the drawer, and building and updating the smaRer iconic display. While the overall program is 
quite large, each Individual updating function is quite compact and manageable. One of the more 
complex methods deals with settings on continuously moveable knobs. Here, there are two active 
regions for the knob (one region on the left to indicate decreases and one on the right to Indicate 
increases). Thus, the necessary geometric calculations must be made to decide the angle from the center 
of the knob to the point moused, redraw the knob setting, and interpret the setting angle in terms of the 
scato of measur*?r*nt to which the knob Is calibrated. 

The schematic diagrams. Another critical pari Of the test station work environment representation is 
the set of schcmviHc diagram. In the real work environment, these schematics are part of the Technical 
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Orders, a muMvolume documentation set. In our task analyses, there was some Indication that finding 
material In the Technical Orders was difficult for some first-term airmen, so we wanted to capture the 
access problem In our work environment simulation. We did this by providing an Indexing scheme for 
screen-based Technical Orders pages that corresponds closely to the Indexing scheme in the printed 
Technical Orders. 

There are two Important characteristics of the Technical Orders that 3houW be noted. First, we made 
paper copies of Sherlock's Technical Orders available to airmen while they used the system. Schematic 
Jagrams are very detailed, and the higher resolution of the print medium makes them easier to examine In 
that form. Second, we developed our own slightly simplified Technical Orders for the tutor. We did this for 
logistics reasons - the complete real ones did not become available quickly enough. In retrospect, this 
may not have been an idea! arrangement, but the success of Sherlock has been demonstrated with post- 
training criterion tasks that used the real Technical Orders, so not much has been lost due to our 
slmpWIcfltlone. 

The reason for also having the schematics available on the computer is that they were the communi- 
cations medium by which the airman indicated the points to which he wanted to attach meter probes to 
make measurements* 

Two Wnds of Indexing were involved In managing Technical Orders content In Sherlock. First, 
there was a reproduction of thw index scheme found in real printed Technical Orders. In any page of this 
index that appeared on the screen, one could access an index entry by mousing Its page number. 
Second, because of the complexity of the schematics and their partial redundancy, 1 1 it was necessary for 
Sherlock to have an Index of test points. This Index made it possible for airmen io determine which 
pagers) of schematics they should consult In order to access any given test point 12 

C arrying out teats. To carry out tests, 13 then, the airman would first indicate In the planning menu that 
he wr tied to test a particular printed circuit board or other component (e.g., a switch). Sometimes, the 
deeir d component first had to be "extended," manipulated to make test points accessible. Then, In most 
cases, he would need to set the controls on a tand-hek) multimeter, the most common piece of test 
equipment The hand-held meter dteplay is another front panel knob-and-dlal simulation of the sort shown 
In Figure 5. After making the appropriate settings, the airman would then call up the appropriate page of 
sche ma tic s , the page on which he could indicate where the meter probes should be placed. The hand- 
held meter then appears above the schematic diagram in iconic fonm, as shown in Figure 7. 

If the requested test has a result that Sherlock knows about, the appropriate Information Is 
displayed in the meter Icon, as shown In the Figure, if not, then the airman Is Informed that the measure- 
ment Is not available, Sherlock Is also able to note cases where the meter Is set Inappropriately. For 
example, there may be a spot where one can appropriately measure resistance but where a voltage value 
might be meaningless, or the meter might be set for DC+ Volts and the probes placed to produce a 
negative voltage reading. In this case, appropriate comments are offered by Sherlock, so that the airman 
doesn't think his decision about where to measure was wrong. 



1 1 A given point In the circuitry may appear on more than one page of the schematic diagrams. 



12 Test points are denoted by a combination of the printed circuit card number and the specific 
connector on that card that is desired. So, to access Test Point 2 on Card A1 A3A1 0, an Airman could look 
up A1A3A10-2. 



13 The discussion which follows concerns tests carried out after a fault Is evident. The routine part of 
the job, which Involves using the test station to test units removed from aircraft is also simulated in Sher- 
lock, but In a different way. Those tests tend to involve switch settings on the front panels of the test 
station, while tests carried out to diagnose an apparent test station failure involve manipulation (e.g., 
placing meter probes) within the circuitry. 
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In Hit current version, we rolled on expert guidance concerning which test values Sherlock should 
know about We provided all test values that our experts believed would be used by experts, along with a 
number of additional test points that the experts expected trainees to request. Some test points were 
IntentoNrty omitted, In order to force the trainee to ask for help rather than succeed with an inefficient 
strategy. For example, u m2y be possible to find a gap in a lengthy circuit path by exhaustive point-to-point 
continuity testing starting at one end of the path and working toward the other, in order to avoid having 
the trainee successful solve the problem in this way, some test values consistent with the sequential 
strategy but Inconsistent with a space splitting strategy were omitted, making the strategy unworkable. 

While this approach was generally successful, some trainees who used the tutor complained about 
"missing" teat points (sometimes because they felt that a purely sequential testing strategy was 
appropriate), and I n d ee d a smaR number of reasonably requested test values were not available. We now 
believe that the key to Improving fto part of Sherlock b First, 
aNpotenlalyrolevaMtostv«km Second, Intelligent coaching should be provided In 

cases where the pattern of tests shows successful but Inefficient testing. 

The procedure we hope to use in our next version for gener a ting test point values will be based on 
the notion of a drouH path that Is active In a given test station configuration. That to, wton the test station 
Is being used to perform a given test on the unit from the aircraft, a path Is switch**) into ptace involving a 
stimulus source which is routed to the unit and a measurement device which is also routed to certain 
outputs of the unit AR of the measurements between consecuthr test points In that path should be 
provided, we now think. Further, H the switching logic Is bad or the controls for It are bad, then measure- 
ments of points In the paths controMng the point at which the primary test circuit Is disrupted must also be 
Included In the collection of known test values. Similarly, paths involving power sources for devices In the 
primary path must be included If Implicated. This would have the effect of making the absence of a test 
virtue an Indication that it was not relevant to a reasonable model of the test station as configured at the 
time of failure. Currently, the absence of a test value Indicates that It is not relevant to an efficient 
diagnostic strategy. 

To a ooomp lsh the second goal, of InteMgent coaching for inefficient testing sequences, we will need 
to include an assessment of testing efficiency In a wrap-up discussion that we would like to add to 
Sherlock. An I n teW gen t coach might consider the length of the trainee's testing sequence, compare it 
to the length of an expert sequence, and then comment on the difference If It Is large. Similarly, a lucky 
guess, which now produces only positive feedback, might be labeled as such In a wrap-up commentary. 

fiOOOtflfcig the work environment to the abstracted problem space-based coach. We have thus far 
described two levels of function in Sherlock. The abstracted problem space level is a hierarchy of plans 
and actions down to the level of actions involving replaceable components: boards, switches, etc. Most 
of the hinting and virtually al of the student modeling is driven from this level. The work environment 
represents a more detailed level, the level of individual tests of small aspects of the replaceable 
compon e nts. At this level, coaching must be more generic or must be the result of data sent back to the 
a b s tracted problem space level of representation. We now turn to the manner in which the two levels are 
linked. 

At the start of a problem, the student is following the Technical Orders (TO), doing specific tests of 
the unit from the aircraft by setting controls on the test station and recording meter readings from it. He 
continues doing this untH he finds a test value that is outside the acceptable range, after which he comes 
under the control of the abetracted-problem-space coach. The testa are conducted using a printed set of 
Technical Orders which specify which switches must be set on the test station and what the appropriate 
teat vriues are. Each test in the TO is represented by a specified object that knows what settings are 
required on the teat station, what value should be displayed on which device to indicate the outcome of 
the test, what button on the test station must be pushed to produce the reading, whether it is In bounds 
or not (I.e., whether the test is passed by the aircraft unit or not), how the student has performed on this 
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test If he has already done It before," and a few other details. The object knows how to decide whether 
doing the test Is currently appropriate, how to let the student conduct the test how to comment if steps 
are left out or an unsafe condition created, and how to provide a resultant meter reading. 

When a fault Is first evidenced, 16 the trainee begins Interacting with a wioer-ranging menu structure 
that permits him to express plans and take high-level actions, swap replaceable units, reseat boards, etc. 
Once he pro p oee e to test a board or similar unit, he is permitted to call up the relevant schematic diagrams 
and choose a measurement device. He then ^*nes under the control of the work environment 
simulation, doing tests and receiving feedback (but stilt having access to hints). He continues doing tests 
as long as he Ikes, eventually indicating that he is doce by mousing a "Done" box on the screen. 

At this point, the work environment executes a small production system that Is able to update the 
a bstracted problem space by marking any parts of the space that have been ruled out by any combination 
of earlier a b st r a cted i xpbfem-spaoo moves and work environment tests . Not only are the nodes that are 
ruled out marked as such, but a message concerning why they are ruled out is sent to them so that they 
have It avaMatois to provide later explanations. Thus, if a trainee later tries to search a ruled-out portion of 
the problem spaoe, he wM receive a message teing him how the proposed action has already been ruled 
out Currently, this message is provided after the action Is taken, Just to advise that It wasn't really 
necessary. For example, in one of our problems, one trainee did some tests that ruled out the unit from 
the aircraft as the source of a fault Later, If he had swapped a card in that unit, this would have been 
permitted, but he then would have received the message: 

You took a voltage reading at A2A3A7 Pin 58 with respect to ground. 
The result of this measurement ruled out this unit 

This production system linkage is necessarily an expert system. That Is, It only contains the rules experts 
give us, since we acknowledge at the outset that deep reasoning at the level of individual circuit values is 
an Intractable task for a system of the complexity of the test station for which our tutor fs designed 

Student Modeling 

Hints can be tailored to a students ability, provided that the coach knows something about that 
studenfs abMty. This knowledge is called a student model . As discussed above, there are two levels of 
student modelng in Sherlock. A competence model records the extent to which various Instructional 
goals of Sherlock have been achieved for a given trainee. This model cumulates over the entire 
experience of the trainee with Sherlock. A second model, the performance model is computed for 
each trainee at the time thai he starts a particular problem. 

Both models are oxfiday models. That is, they Involve a set of goals or actions for each of which an 
estimate of student capability is recorded. The competence model is based on a curriculum hierarchy 
which lays out the goals of Sherlock. For each goal or subgoal, a notation is made concerning the level 
of capabMty the student is currently thought to have. Four levels are used: unlearned, perhaps , probably . 
andatCDOO- A capability at the padiapa level has been apparently manifested recently but Is thought to be 
very fragile. If K falls to be manifested in the next few cases where It Is appropriate, it will revert to the 
unlearned state. If K is remanifesteo, It will move to the perhaps state. If It Is regularly and reLably 
manifested, K wll move to the atCQOfl stale. In addition, Sherlock also keeps the history of evidence 
used to determine the state for each curriculum subgoal . 
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14 Normally, a test Is repeated after a swap, to be sure that the system has been fixed, so multiple 
records of student performance on a test may be recorded. 

1* The trainee actually starts in the mode of using the test station to diagnose parts from an aircraft. 
This part of Sherlock will be described below. We are concerned in the present section with what 
happens once a faun is evidenced. 
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When a trainee begins a problem, a performance model for him Is computed. For each node of the 
A b s tr a cted problem space, an estimate Is made of how well he is expected to do at that node. This 
estimate Is computed by the node object in the following manner. First the node object takes note of 
which curriculum subgoab are relevant to performance at this node. These objects are listed hi a "slot" of 
the node object For each subgoal on the list, the student model entry for that subgoal Is noted. Based 
on these entries, expected trainee performance on the node is recorded as good, okay, or bad. After the 
problem Is completed, Sherlock checks to see whether actual student performance at each node 
matched expected performance. Exceptions to expectations result hi changes to the competence model 
for the student 

Hlnta 

The coaching provided by Sherlock consists primarily of hints to the trainee as he tries to solve 
each fault Isolation problem. Some hints are generated dynamically (especially the recapitulation hint 
d escri b ed below), whNe moat are "camieo.* Except in a few specific cases, hints are provided only in 
response to a trainee's request The primary exceptions are when the student has finished taking an 
action In a problem-space node and now wants to move to another point In the problem space that Is 
definitely a wrong choice. On certain occasions, In the Judgement of our expert consultants, certain 
choices were so dearly inappropriate that it makes sense to ted the trainee this rather than letting him 
dbcovor ft for himself. In these cases, making a bad choice of what to do next prompted a hint explaining 
why It was a bad choice. 

A second kind of case Is when the trainee wants to redo a step that has already been explicitly taken, 
such as sw a pp i n g a board again.™ in this case, Sherlock wNI hitenupt with a message that the proposed 
activity has already been completed. A reined case gets different treatment. H the activity associated with 
a node in the problem space has been rendered redundant because some combination of other actions 
has logically ruled It out (e.g., several tests of other components might produce results that logically entail 
the correct function of a Afferent board, which one would then have no reason to swap or test), then the 
traine e Is stt alowed to do the activity, but after he is done with K, he is advised that it was not necessary, 
and told why* 

The remaining hints are provided In response to requests for help or pressing the panic button. 
When the panic button is pressed* the trainee to problem 
space that are stM candidates for addition* activity. Then, he is given the Initial planning menu that Is 
available at the time a fault is first encountered, essentially being allowed to start over again, from the top, 
on what Is left of the problem. Our thought was that the one occasion on which we could probably get the 
trainee to step back from the details of tactics to look at strategy broadly was when he was completely at a 
toes for next actions. 

When a hint is requested by mousing the UfilB button on the screen, then the system makes a 
decision on how to respond based on the performance model of the student for the problem space node 
at which he is currently located and his activity status within the node. There are four categories of activity: 
Action. Outcome, Conclusion and Option, AtfiQQ refers to deciding which test to carry out and how that 
teat Is done. Outcome refers to getting the results of a test perhsps by rearing a meter. Conclusion 
refers to the determining the m ea n i n g of the results, and flnMga refers to deciding what to do next. So, 
for example, one might enter the problem space node for testing a particular printed circuit board but not 
know what to measure on that board. Astdng for help at that point would produce an Action hint On the 
other hand. If one has just made an oscilloscope measurement, then presumably the needed help is in 
the area of Outcome or Conclusi on. If one has already closed the page of schematics for the board and 
then asks for help, an Qjtfton hint Is provided. 



1* in Sherlock, all replacement components always work. This is not entirely realistic, but is part of 
the general strategy of focusing trainee time on acttvtty deemed most Hkely to Improve basic 
troubleshooting sMI. 
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Several structural (totalis can help assure that an appropriate hint Is provided. First, some stages of 
activity wNtrtn a problem space node are well marked, especially the Action and the Option stages. 
Second, as available hints at one level are exhausted, hints for the next level can be provided. So, if one 
gets some Action hints and keeps asking for more help, eventually Outcoms. Conclusion, and Option 
hints would also be provided. We planned a third approach, In which Sherlock was to put up a menu 
asking the trainee what kind of help was needed, but this plan was not Implemented, due to scheduling 
urgencies. 

Once the type of hint to be delivered is determined, then a decision about how explicit a hint to 
provide must be made. Hints are classified from 1 to 5 on a scale of explicitness. Hint level 1 is always a 
recapitulation of what the trainee has done so tar in trying to solve the problem. 17 Hint levels 2 to 5 are 
assigned to pre-writtsn hints that were produced by our subject matter experts. There is not always a hint 
amiable for each level and each type. 

The first time a trainee asks for help on a node, he gets a Level 1 recapitulation hint. The next time, 
he gets a hint d e termined by his expected performance on the node at which he Is currently located. If 
the performance model says that he should do a Good Job, he next gets a Level 2 hint. An Qkax rating 
produces a Level 3 hint, and a Bad rating gets a Level 4 hint (see the discussion of the Performance 
Model above). Further requests result in progressively higher level hints tin the appropriate set is 
exhausted, followed by hints from successive type groups (e.g., if Sherlock is out of conclusion hints, it 
gives an option hint). 

Some General issues 

Situated teaming. The ideal of "situated learning" Is currently fashionable, especially In informal 
discussions of Instructional systems by their designers. It has been noted (Rogoff & Lave, 1984; Jordan, 
1967) that In simpler, more unified and coherent cultures, much of learning Is socially supported by 
groups of people In direct contact with the parts of the environment in which the knowledge will be 
needed. People learn about detals of agriculture by watching planting, cultivating, and other processes 
and helping wNh them. TaHors learn to make clothes in an apprenticeship. Often, children follow In their 
parents 9 footsteps, and many Jobs are done at home, or otherwise with children present, so one lives in 
the environ m e n t one wi act in, and one watches and helps with expert actions. Learning does not seem 
to be a problem when al of these conditions are met. 

Life in a large, pluralistic, high-technology society tends to lack all of the characteristics just listed, and 
there have been proposals for forms of "cognitive apprenticeship" that attempt to provide some of the 
support that is automatically present in more primitive societies (cf. Coffins & Brown, 1988). We have tried 
to design Sherlock with these Ideas In mind. So, for example, the forms of work match those on the real 
job. Each front panel with knobs and dials looks like the panel in the real work environment. The 
schematic diagrams, Instructions for testing units from aircraft, and other documentation, while slightly 
simplified, retain many aspocts of those used in the real job: the symbols in diagrams are the same, the 
layout of testing algorithms is the same, the indexing schemes are the same, etc. 

What Is not yet evident Is whether this approach matters. The experiments have not yet been done 
to compare our approach to alternatives. There are broad alternatives, such as componentiaf rather than 
holistic training. In addition, there are a number of detals of realism or abstraction that might or might not 
be Import an t For example, tests might be requested via a menu or calculator keypad in which test point 
numbers and measurement details (volts vs. ohms, etc.) were requested. We might not require people to 
put the simulated test station Into a safe condition via knob settings before making measurements. We 
might even go to diagnosis of arbitrary circuits different from those used on the job. Transfer studies will 
have to be conducted to fully justify our approach For now, all we know Is that, in general terms, it works 




17 it differs from a Panic hint in that it is a structured account of the actions taken, while the Panic hint 
Is a structured account of the problem space portions remaining to be searched. 
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very wel and that airmen are able to start using Sherlock with minimal introduction and without having to 
Invest much thinMng capacity into figuring out how to talk to It. 

iDdhlfclUJaiLflQMhlaB A second area worthy of comment Is Individualization. In mo9t Individualized 
Instructon systems, and even In the early plans for ours, Individualization is via rate of progress through a 
curriculum. There Is an Implicit ranked listing of learning tasks, and people are passed over various of 
these as a (unction of the system's knowledge of their competence. We took a different tack. All of our 
trainees get the same series of problems. What differs with differing competence levels is the specificity 
of support (through hinting) that they receive. A fast learner will characteristically be given very general 
and sketchy help unless he keeps asking for more, while a slow learner, at points where he is out of his 
league, wM almost be waked through the problem solving process. 

TNs approach arose after one of us spent a week evaluating uses of individualized print and 
comp u ter tec hnology at schools In Israel. 1 * He observed schools in which there were three different sets 
of reedkig material, but al with the same stories in them. The fast track received versions of the stories that 
were longer and more detailed, and the corresponding workbook assignments were also more sophisti- 
cated and demanding. At the slow end, the stories were mostly gist with simpler words and simpler 
workbook tasks. Nonetheles s , In class dbcusstons, the whole class could talk about the same story and 
lis imptcaBons. 

" We skriarty thought that one way to leverage the training provided by Sherlock was to provide 
common experiences, so that discissions at the end of the day could be about problems and how they 
were solved rather than about who was ahead of whom. We do know that Sherlock generated a lot of 
excitement at the bases where ft was tested, and we Ike to think that (a) that excitement Is Important to 
effective le a rni ng and training; and (b) that our individualization approach contributed to that excitement. 
However, further analysis of the data from our field tests and possibly further experimentation will be 
required to confirm our views. 

On deciding what works. Where we now stand is that we have a coached practice environment that 
produoes effects of clear economic Importance. However, it will take considerable study to determine 
which of the features of our design worked, which were exercised often enough to even have a chance of 
being Influential, and which, while possibly Important in our thinking, need further and more direct 
demon s tration and evaluation. This is the nature of any artifice with utility. It reflects our hypotheses and 
provides piefimlna iy pilot testing of the validity of those hypotheses, it is, in consequence, both more and 
less than the theory that motivated it 
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Tablet: Abstracted Problem Spare Object Structure 

InetanceName - What this hoda wil ba called In action menus and submenus 

Massage • Printed In the Concept Window when the student enters the node. If this node is a part 
of the problem space that K doesnl make sense to search, this message may amount to a hint thai 
this is the wrong place to look. 

Display - Which view of the work environment should be displayed when this node is entered; 
defaults to leaving current dtoplcy in window. 

Preconditions - Conditions to be met upon entry to the node (for example, If this node represents 
swapping a drawer of the test station, that drawer must be turned ofl before the swap can occur. 

Options -List of nodes to be Included In the actions mefu for this node; I.e., what cm come next? 

Parana • Backward pointers in abstracted problem space graph structure. 

unuurvn • rorwara pontore. 

CurricukimSubGoals - List of relevant curriculum Issues for this node. These Issues are used to 
calculate the student performance expect a tion for the node and therefore control the hints that the 
student gets. 

Marker - Indicates whether the trainee completed the search actions subsumed by this node. Can 
also contain addMonal Information about circ u mstances in which entering this node Is nonproductive 
or inappropriate, Inducing messages for dbpiay In those circumstances. 

Hints - Hints for the node (prewritten hints; additional hinting is generated dynamically). In a plan 
node this Is a simple 1st which are pross n ts dl n turn to the student hi an actions node, there are four 
categories of hints and as many at four hints for each category. 

Outcome • PASS, FAIL or In some nodes a list of conditions which determine outcome of 
measure m en t/ tost actions subsumed by this node. 

NumberOfHlMsUsod - Keeps track of the number of hints used (hi the action nodes, by 
category). 

LaatHlntUsed • The number of the last hint used (in action node, by category). 

WhoCalledMe - Node the trainee entered from. 

WnereDfdlGo • Node the trainee exits to. 

NodeHlatory • List of student actions in the node with time stamps. 

StudentPerfRatlng • Actual (as opposed to predicted) student performance rating at the node. 

Student PerfExp • Expected student performance at this node. 

LevellnNode • Controls the category of hint given (e.g., taking measurement actions, Interpreting 
outcomes, and deciding what to do next). 

RunTOTestFIg • T If the student needs to rerun the TO test (the activity of the test station that 
failed because the test station was broken) at this node. 

SequenceNumber - Sequence number for order of node entry by trainee. 
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